From idealization of methods to conscious practice
2025-12-02
Have I ever considered multiplicity? No 😔
When testing a null hypothesis H_0 against an alternative H_1, we can make two types of errors.
| Reality | Reject H_0 | Fail to reject H_0 |
|---|---|---|
| H_0 true | Type I Error (False Positive) | Correct decision |
| H_1 true | Correct decision | Type II Error (False Negative) |
We (usually) control the Type I error rate at a pre-specified level \alpha (typically 0.05).
You compare the scores of the control and experimental groups and obtain a p-value of 0.05.
If, hypothetically, the experiment were repeated a great number of times when the null hypothesis is true, we would falsely reject the null hypothesis about 5% of the time.
After comparing the scores of the control and experimental groups, you find that the posterior probability that the groups differ is 95%.
Based on the data and your prior, there is a 95% probability that groups differ. If you act as though the groups truly differ, there is a 5% chance that this decision is wrong.